NORTHWESTERN UNIVERSITY Using Analogy to Model Spatial Language Use and Multimodal Knowledge Capture A DISSERTATION SUBMITED TO THE GRADUATE SCHOOL IN PARTIAL FULLFILLMENT OF THE REQUIREMENTS for the degree DOCTOR OF PHILOSOPHY Field of Computer Science By

Using Analogy to Model Spatial Language Use and Multimodal Knowledge Capture Kate Lockwood Language and knowledge capture are two skills that allow us to create structured representations of the physical world for reasoning and communication. Spatial prepositions are a form of specialized language that is used to relate two objects in space. In addition to communicating the static location of objects, spatial prepositions contain layers of information about the interactions and potential interactions between objects, agents, and their environments. While a large amount of information is encoded in spatial prepositions, the components of scene that contribute to their use are only a fraction of the possible features available. In order to learn the correct spatial prepositions categories in a language, a learner must figure out how to abstract the important concepts without being distracted by surface features. When communicating about complex spatial relationships, a diagram can often, as the saying goes, “be worth a thousand words”. Diagrams can communicate complex spatial concepts concisely, which is why they are often used in educational materials. Information from text accompanying diagrams must be integrated with the spatial information from the diagram to create a cohesive understanding of the concepts being communicated. This process is called multimodal knowledge capture. It is multimodal because the information being captured is presented in two different modalities: text and diagrams. Often spatial language, in particular spatial prepositions, in the text provides clues about how to best integrate the different representations. This dissertation addresses computational modeling of both learning spatial prepositions and multimodal knowledge capture. In particular, it examines the role that structure-mapping plays in both tasks. In spatial preposition use, sequential generalization over multiple scenes results in the 4 abstraction of core category concepts. In multimodal knowledge capture, structure mapping provides a framework for integrating structured representations from different modalities.